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Abstract 

Safety is an important element of dependability. It is 
defined as the absence of accidents. Most accidents involv- 
ing software-intensive systems have been system accidents, 
which are caused by unsafe inter-system or inter-component 
interactions. To validate the absence of system hazards 
concerning dysfunctional interactions, industrials call for 
approaches of modeling system safety requirements and 
interaction constraints among components. This paper pro- 
poses such a formalism, namely interface control systems (or 
shortly C-Systems). An interface C-System is composed of 
an interface automaton and a controlling automaton, which 
formalizes safe interactions and restricts system behavior at 
the meta level. This framework differs from the framework of 
traditional model checking. It explicitly separates the tasks 
of product engineers and safety engineers, and provides 
a top-down technique for modeling a system with safety 
constraints, and for automatically composing a safe system 
that conforms to safety requirements. The contributions of 
this work include formalizing safety requirements and a way 
of automatically ensuring system safety. 

1. System Safety Requirements 

Critical systems are always controlled by software appli- 
cations, which overcome the shortcomings of human control, 
but also introduce new failure modes that are changing the 
nature of accidents 11]. Inter-system and inter-component 
dependability are becoming important, since industrials are 
developing complicated software-intensive systems which 
consist of numerous components (subsystems) and a huge 
number of actions (both internal and interactive). A recent 
challenge of dependability is the system accident, caused by 
increasing inter-system and inter-component couplings and 
their interactive complexity ||2]ll3l- In contrast, accidents aris- 
ing from component failures are termed component failure 
accidents. 

System safety and component reliability are different 
elements of dependability. They are system property and 
component property, respectively [2|. Reliability is defined 
as the capabiUty that a component satisfies its specified 
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Figure 1 . A Chemical Reactor Design 



behavioral requirements, whereas safety is defined as the ab- 
sence of accidents — events involving an unacceptable loss 
ID . People are now constructing intellectually unmanageable 
software systems that go beyond human cognitive limits. 
This allows potentially unsafe interactions to be undetected. 
Accidents often result from hazardous interactions among 
perfectly functioning components. 

As an example, a system accident occurred in a batch 
chemical reactor in England |5|. The design of the system 
is shown in Fig. [T] The computer controlled the input flow 
of cooling water into the condenser and the input flow of 
catalyst into the reactor by manipulating the valves. The 
computer was told that if any component in the plant gets 
abnormal, it had to leave all controlled variables as they were 
and to sound an alarm. On one occasion, the computer just 
started to increase the cooling water flow, after a catalyst had 
been added into the reactor. Then the computer received an 
abnormal signal indicating a low oil level in a gearbox, and 
it reacted as its requirements specified: sounded an alarm 
and maintained all the control variables with their present 
condition. Since the water flow was kept at a low rate, then 
the reactor overheated, the relief valve lifted and the contents 
of the reactor were discharged into the atmosphere. 

Some other system accidents in avionics are also due 
to uncontrolled interactions between components |6|. The 
self-destructing explosion of Ariane 5 launcher was resulted 
from the successive failures of the active Inertial Reference 



System (IRS) and the backup IRS ID. Ariane 5 adopted the 
same reference system as Ariane 4. However, the profile 
of Ariane 5 was different from that of Ariane 4 — the 
acceleration communicated as input value to IRS of Ariane 
5 was higher Furthermore, the interactions between IRS and 
other components were not redefined and checked. Due to 
the overflow of input value computation, the IRS stopped 
working |7l. Then, the signaled error was interpreted as a 
launcher attitude, and led the control system to rotate the 
tailpipe at the end stop (SJ. 

In these accidents, the components are reliable in terms of 
satisfying their specified requirements, but the systems are 
not safe as a whole. Since most software related accidents 
have been system accidents [Fl, people need to model and 
constrain interactions of system components to validate the 
absence of dysfunctional interactions. As Leveson men- 
tioned in STAMP (Systems-Theoretic Accident Model and 
Processes) [1 J, these accidents result from inadequate control 
or enforcement of safety-related constraints of the systems. 

Traditionally, in order to validate the absence of sys- 
tem hazards, industrials identify system safety requirements 
ll9l lfT0l . and use model checking to verify if system behav- 
iors conform to safety requirements fTTl. 

In this paper, we will consider safety requirements as 
control structures that restrict system behaviors at meta- 
model level. That is, the two models of a system and its 
safety constraints will be developed at the same time. Then 
the two models are "combined" to deduce a safe system. 

This paper is organized as follows: the architecture of our 
approach is presented in Section 2. To illustrate the idea, 
a preliminary introduction on interface automata appears 
in Section 3. The interface C-System based on controlling 
automata is introduced in Sections 4 and 5, where examples 
are used to illustrate how to formalize safety rules and 
combine them with a system specification. In Section 6, we 
compare our work to classic verification techniques, such as 
model checking, and conclude the paper 

2. The Architecture of our Methodology 

The most popular technique of system safety verification 
is model checking 1121 . Hundreds of checking patterns are 
collected for system engineers (131 and specific uses in 
safety engineering |11|. In this framework, we have three 
steps in verifying a system. At first, we formalize system 
behavior as a model (e.g., a finite-state transition system, a 
Kripke model 1 14|). At the second step, we specify the safety 
constraints that we aim at validating using temporal logics 
|13|. At the third step, a certain checking algorithm is used 
to search for a counterexample which is an execution trace 
violating the specified features. If the algorithm finds such 
a counterexample, we have to modify the original design to 
ensure safety constraints. 



Unlike model checking, our architecture takes another 
way. It consists of the following steps: 

(1) Modeling system behavior, including specifications of 
its components, internal and external interactions, e.g., using 
interface automata. 

(2) Modeling system safety constraints using a certain 
formal technique, e.g., controlling automata in this paper. 

(3) Combining these two models to deduce a safe sys- 
tem model, that is, a system model whose behavior is in 
accordance with its safety constraints. 

As |i5| mentioned, system behavior specifies an opera- 
tional semantics, which defines what a system is able to do. 
System behavior modeling is achieved by product engineers 
(designers), such as programmers and developers. In the 
example of the chemical reactor control system, the actions 
"opening the catalyst flow", "opening the cooUng water 
flow" and "sounding an alarm" are actions of the system 
behavior 

In the second step, the model of safety constraints spec- 
ifies a correctness semantics, which defines what a system 
is authorized to do. This process is the duty of safety engi- 
neers whose responsibiUty is to assure system safety. Safety 
engineers may consist of requirement engineers, testing 
engineers, managers from higher socio-technical levels who 
define safety standards or regulations |1|, etc. In the example 
of the chemical reactor system, the constraint "opening the 
catalyst flow must be followed by opening the cooling water 
flow" is an instance of system safety constraints. 

In the third step, in order to ensure system safety, we 
combine a system model with its safety constraints model. 
Then we ensure that the system is safe under the constraints 
specifying safety requirements. A precondition of this ap- 
proach is that we must formalize safety requirements. And 
we also need to carefully define the composition of a system 
model and its constraints model. We will introduce such 
means based on controlling automata. 

3. PreUminary: Interface Automata 

To model component-based concurrent systems with dif- 
ferent input, output and internal actions, the theory of inter- 
face automata |16| extends Input/Output automata |17||18|, 
which extends classic automata theory ||T9ll . 

Unlike I/O automata, an interface automaton is not re- 
quired to be input-enabled (i.e., some inputs may be rec- 
ognized as illegal in some states) and only allows the 
composition of two automata (I/O automata allow the com- 
position of infinite automata), and a synchronization of one 
output and one input action results a hidden action after the 
composition. 

Definition 1: An interface automaton (simply an au- 
tomaton) is a tuple A = (Q, S^, S^, (5, S), where: 

(1) Q is a set of states. 

(2) S^, E*^, are pairwise disjoint sets of input, output 



and internal actions, respectively. Let E = |J S*^ |J 
be the set of actions. 

(3) (5 C Q X E X Q is a set of labeled transitions. 

(4) 5 C Q is a set of start states, where IS*] < 1. □ 
In the graph notation, a transition : {q,a,q') G S 

is denoted by an arc from q to q' labeled pk : a, where 
Pfe is the name of the transition. To discriminate explicitly 
the different sets of actions in diagrams, we may suffix a 
symbol "?", "!" or ";" to an input, output or internal action, 
respectively. 

The composition of two composable automata allows 
the automata to synchronize on shared actions, and asyn- 
chronously interleave all other actions. 

Definition 2: Two interface automata A and B are com- 
posable if n Es = 0, E^ n = 0, E^ n Eg = 0, 
Ef n Ea = 0. We let shared{A, B) = T.Ar\Y.B. □ 

Definition 3: If A and B are composable interface au- 
tomata, their product A ® B is the interface automaton 
defined by 

(1) Qa(s,b = Qa X Qb 

(2) E;^^B = i^A U ^b) - shared{A, B) 

(3) EO^s = (Eg u Eg) - shared{A, B) 

(4) E^^s = E;^ U Ef U shared{A, B) 
(5) 

5a®b = { Pi : ((w, m), a, («', u)) \ pi : (u, a, i;') £ 5a 

Aa ^ shared{A, B) Au e Qb} 
U {pj : ((u, u), a, (w, u')) | : {u, a, m') G (5_b 

Aa ^ shared{A, B) Av € Qa} 
U {py : ((w, m), a, (w', w')) | Pi : {v, a, w') G Sa 

Apj : (m, a, u') G (5_B A a G shared{A, B)} 

(6) = -Sa X 6*3. □ 

Note that the name of the transition pij of A(^ B may 
contain the names of two original transitions pi G 6a and 

P] e Sb- 

In the product A^ B, there may be illegal states, where 
one component is able to send an output a G shared{A, B) 
and the other is not able to receive a. 

The composition of two interface automata A, B is ob- 
tained by restricting the product of the two automata to the 
set Cmp{A, B) of compatible states, which ai-e the states 
from which there exists a legal environment that can prevent 
entering illegal states. 

Definition 4: If A and B are composable interface au- 
tomata, their composition A\\B is the interface automaton 
defined by 

(1) QA\\B = Cmp{A,B) 

W '^A\\B ^ ^A®B 

(5) Sa\\b = 6a®b n {Cmp{A,B) x E^n^ x Cmp{A,B)) 

(6) Sa\\b = n Cmp{A, B). □ 



4. Safety Constraints on a Single Component 

In this section, we start from a simple case - modeling 
safety constraints on a single component. In the example 
of the batch chemical reactor (C.f. Fig. [TJ, the computer 
system behavior is modeled using an interface automaton 
A of Fig. 12 1). The automaton A includes a set of input 
actions E^ = {1} (low oil signal), a set of output actions 
E*^ — {c, w, a} (opening catalyst flow, opening water flow, 
sounding an alarm, respectively), and a set of internal actions 
E^ = {e} (ending all operations). 

The normal operational behavior includes opening the 
catalyst flow ipi), then opening the water flow {P2), etc., re- 
sulting in an infinite execution trace piP2PiP2---- To respond 
to abnormal signals as soon as possible, the states qg, qi both 
have a transition labeled which leads to a state that can 
sound an alarm (ps) and stop the process (pg). Unfortunately, 
this design leads to hazardous behaviors: {cw)*clae, that is, 
after a sequence of opening catalyst and water flows (cw)*, 
then the catalyst flow is opened (c) when an abnormal signal 
is received (0, then an alarm is sounded (a). So water is not 
added after the catalyst flow is opened. This sequence of 
events leads to the accident mentioned in Section 1. 

Note that this hazard is due to the uncontrolled sequences 
of transitions — pi must be followed by p2 and not by p4. 
To solve this problem, we need to specify the authorized 
sequences (satisfying safety constraints) on the transitions 
6 and not on the actions E. Thus, these constraints are 
not at the behavioral model level, but at the meta-model 
level. We propose the concept of controlling automata to 
formalize safety constraints. Then, we combine a controlling 
automaton with the system automaton. 

Definition 5: A controlling automaton A over an inter- 
face automaton A = {Q, E, 6, S) is a tuple A — {Q, E, 6, S), 
where: 

(1) Q is a set of states disjoint with Q. 

(2) E is a set of terminals, such that E = (5. 

(3) ^ C Q x E x Q is a set of labeled transitions. 

(4) S* C Q is a nonempty set of start states. □ 
Note that the transitions S of A are terminals of A, so we 

say that A is at the meta level of A. Figure [3] illustrates the 3 
levels in our framework. Let E* be a set of execution traces 
of actions, A describes the behavior on E. A specifies the 
behavior on the A-transitions (E — S), that is, a behavior 
on the behavior of A. This meta-behavior expresses safety 
requirements. 

In the example, to prevent accidents, we need to impose 
the safety constraint "opening catalyst must be followed by 
opening water," that is, "whenever the transition pi : c 
occurs, the transition p2 : w must occur after that". This 
constraint can be formalized as a controlling automaton A 
of Fig. |2j2). When we express this constraint, we only 
specify the sequence of transitions pi , p2 at the meta-model 
level, and we concern little about the implementation of 
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Figure 2. Automata of the Reactor Control System 
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Figure 3. A 3-levels Overview 



the system at the model level. The next step is to compose 
the system automaton A with its controlling automaton A, 
and automatically generate a system C satisfying the safety 
requirement. 

Definition 6: The meta-composition C of an interface 
automaton A = {Q, S, S, S) and a controlling automaton 
A = (Q, E, S, S) over A is a tuple: 



C = A^A = {QxQ, S, 5', S xS) 



(1) 



where pk : {{qiAj), a, {qmAn)) e 5' iff, 

(1) Pk ■ {qi,a,qm)^(^ S, and 

(2) {qj,Pk,qn) e ^■ 

We say that A and A constitute an interface control 
system (or simply interface C-System). □ 

The symbol ~^ is called meta-composition operator, and 
read "meta-compose". Its left and right operands are an 
automaton and a controlling automaton, respectively. No- 
tice that an interface C-System is equivalent to the meta- 
composition C of an interface automaton and a controlling 
automaton. 

Notice that 6 ~ {pk}kG!C plays a key role in associating 
transitions of A and terminals of A. For our example, we 
combine the automata A and A of Fig. |2] thus we get the 
automaton C = A^A of Fig. |4] where qij denotes {qi, qj). 

The meta-composition contains exactly all the paths satis- 
fying the safety constraint. Formally, we have the following 
theorem (the proof is omitted for its simpleness and intu- 
itiveness from the definition): 

Theorem 7: Given A, A and the meta-composition C, 
an execution trace e S* is recognized by C iff, is 
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Figure 4. Tine Meta-Composition C 



is recognized by A, and its transition trace t(, G 5* is 
recognized by A. □ 

Obviously, the set of traces of C is a subset of the traces 
of A. Formally, let L(A) be the set of traces of A (i.e. the 
language of A), we have L(C) C L(A). 

Thanks to A, the hazardous execution traces, for example 
cwclae, which exists in A, will be eliminated, because its 
transition trace P1P2P1P4P5P6 ^ L{A) (the language of A). 
The comparison between A of Fig. and C of Fig. |4] 
highlights the hazardous transition p4 of A. However, in 
general, this diagnosis is much more complex and cannot 
be achieved manually, since a real system A has too many 
states to be expressed clearly on a paper. That is why we 
developed a formal and automated method for eliminating 
hazardous transitions. 

5. Safety Constraints on Multi- Components 

To illustrate this case, we use an example concerning 
a system composed of two components with interactions: 
a candy vending machine and a customer We hope that, 
since this class of examples is so popular in the literatures 
of formal methods (e.g., Hoare's Communicating Sequential 
Processes (CSP) and I/O automata ifTTll ). they will provide an 
interesting illustration of our idea. The candy machine A,n, 
specified in Fig. may receive inputs foi,&2 indicating 
that buttons 1 and 2 are pushed, respectively. It may output 
s, a, indicating candy dispensation actions, SKYBARs and 
ALMONDJOYs, respectively. The machine may receive 
several inputs before delivering a candy. A greedy user Au, 
specified in Fig. |32), can push buttons bi , b2 or get a candy 
s,a. The greedy user does not wait for a candy bar before 



pressing a button again. 

The composition of the machine behavior and the user 
behavior is defined by A,nu = ^mll^u of Fig. |33), where 
Qij denotes the composite state {mi,Uj), pij denotes two 
synchronized transitions {pi,pj}. A transition of the compo- 
sition may be composed of two transitions of components. 
For example, : s is a synchronization of pi : si 

and P13 : s?, which belong to and A^, respectively. 
Generally, a transition of A = P\\Q may be composed 
of one or two transitions of its components, where two 
transitions constitute a synchronization. 

In the context of meta-composition, a composite transition 
is allowed if and only if both of its sub-transitions are 
allowed by its controlling automaton. Thus, we define the 
meta-composition operator as follows: 

Definition 8: The meta-composition (or interface C- 
System) C of a composition A = P\\Q and a controlling 
automaton A = (Q, E, 6, S) over A is a tuple: 

C ^ A^A= {QaxQ,J:a,S\Sax S) (2) 

where px : {{v,u,q),a,{v\u' ,q')) S S' (pj contains a set 
of transitions {pk}kei) iff' 

(1) Px : {{{v,u),a, {v',u')) G Sa, and 

{2)\fk:kGX»{q,Pk,q')€S. □ 

Notice that the specification of the example allows a 
hazardous situation: the greedy user repeatedly pushes the 
buttons without giving the machine a chance to dispense a 
candy bar (e.g., the transition labeled P5,ii : 61 of qn does 
not allow the transition {qu, s, qoo) to be fired). To prevent 
this situation, the following constraints forbid successive 
occurrences of pressing buttons: "the transitions pii,pi2 are 
not allowed, when interactions occur between the machine 
and the user". Differing from the previous example, this 
type of constraints needs to synchronize the actions of the 
machine and of the user 

Formalizing the constraints, the semantics of the control- 
ling automaton Ac of Fig. |6j 1) is: whenever the user pushes 
a button ipQ,pio), she or he cannot push it again {pii,pi2), 
but can only wait for a candy bar. 

Combining the whole system A„iu with its constraint 
Ac, we get the system C = (A„i||A„) • Ac in Fig. |6j2), 
where qijk denotes the composite state {mi,Uj, Ck)- All of 
its execution traces satisfy the constraint, and thus prevent 
the hazardous situation. 

Since we formally defined the meta-composition operator, 
it can be easily implemented to be an automated tool. Thus, 
it can be applied to more complex systems. 

6. Conclusion 

We proposed formalizing system safety requirements us- 
ing controlling automata. As we illustrated using examples, 
this approach can formally model safe interactions between 
components or systems. This framework differs from the 



one of model checking. It explicitly separates the tasks 
of product engineers and safety engineers, and provides a 
technique for modeling a system with safety constraints, and 
for automatically composing a safe system that conforms to 
safety requirements. 

The essential ideas of our approach are the separation 
and formalization of the system specification A (core func- 
tional requirements) and the safety constraints A (safety 
requirements). The automaton A handles inputs to produce 
outputs using activities depending on the states, whereas the 
controlling automaton A treats activities to produce the set 
of acceptable activities depending on safety requirements. 

Our framework has different objectives and uses different 
approaches to those of model checking. Model checking 
techniques use a bottom-up approach — it verifies execution 
traces E* at the lower level Li to prove the correctness and 
safety of the system model A at the middle level L2 (see 
Fig. [3]l. However, our proposal uses a top-down approach 
— we model safety requirements as acceptable sequences of 
transitions {5*) at the higher level L3 to ensure the correct 
use of A. Then any execution trace (at Li) that conforms to 
the meta-composition C is definitely a safe execution. The 
two techniques are complementary. Model checking may be 
used to reduce the design fault likelihood, and our approach 
can be applied to avoid behavior that are not in accordance 
with some critical safety requirements. 

This paper continues our work on C-Systems (formal 
language control systems). In ll20l . we actually proposed the 
input/output C-System. The context-free C-System was pro- 
posed in [21 1 for restricting the use of modeling languages, 
in order to ensure guidelines and consistency rules of UML. 

In the future, it might be a good direction to study the 
formalization of parameterized safety constraints. Another 
direction is empirical case study on applying this formalism 
in large and complex systems. 
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